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Digital First Assessment: mid-Beta 


Ministry: MGCS 
Digital product/ service: DA-690, ServiceOntario Driver and Vehicle 
Conversational IVR 


The objective of the Beta phase is to build a working version of the service based on your alpha prototypes. 


In beta, your team would: By the end of beta, you are expected to have: 
e maintain a prioritized list of the user stories that have e a fully functional version of the service that 
already been researched adequately meets user needs in the live environment 
e build a minimum viable product that can be used by e alist of major bugs and usability issues that have 
the public in a live production environment been identified and fixed 


e continuously test the service with users to collect 
feedback and discover helpful insights 

e test the MVP for accessibility and use assistive 
technologies like screen readers with users in your live 
environment 

e measure the service against key performance 
indicators 

e release updates to patch the service 

e release updates to improve the service, implementing 
new features in response to user feedback 

e resolve any remaining technical or process-related 
challenges 


e a backlog of features to investigate with users and 
complete in order to improve the service 
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Our services are of fundamental importance to people's everyday experiences, livelihoods, and health. The goal of the Digital 
First Assessment (DFA) is to help ministry teams develop services in alignment with the Digital Service Standard. Together, we 
create the conditions where simpler, faster, better government can become a reality. For more information on the DFA 
process check out our intranet page: Ontario.ca/DigitalAssessments. 


Your answers to the questions below will help our panel of assessors determine where your team may benefit from ODS 
supports. If you don’t have an answer or your team hasn't completed some of the work we've referenced, that’s ok! Just let us 
know how you plan to incorporate it in the future. Your honest responses in plain/ conversational language allow us to better 
understand your challenges and help us identify creative ways to support your work. 


The Assessment has questions under the following sections: 


Understand users and their needs 


Establish the right team 
Be consistent 


Design the service from start to finish 
Ensure users succeed the first time 


Test the end-to-end service 


Make it accessible 


Be agile and user-centred 
9. Use open standards and common platforms 


St a Or oe N a 


10.Embed privacy and security by design 


11.Support those who need it 
12. Measure performance 


13.Be a good data steward 
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The Assessment 


Note: If you would like to share artifacts to demonstrate your work, please share these documents as email attachments to your 
submission email. (NOT as hyperlinks from your phase form). 


Previous DFA (if any) 


Has this initiative been through a Digital First Assessment before? Yes - the last assessment note November 30, 2022, with 
results of 9 out of 11 items met and determined that a Mid-Beta report would be submitted next. The team has been working 
closely with ServiceOntario, with focus on the recommendations for content, design the service from start to finish, and 
measure performance. 


e The focus of this has been work on Content, with functionality, research, approvals and tapping into that process with 
CMI and MPBS Comms. 

e The team has been working to create the technology to enable Conversational IVR capability with AWS 

e There has been progress since November with recent research, establishing Change Management & Implementation 
in Customer Care with the project and operational responsibility of the go forward team with ServiceOntario, 


o CMI is leading the approval process of content, any further development and organizational readiness with the 
support of DEEP 
o ACOE consulted — No testing required 
o Privacy consulted — No PIA required 
If yes, please list the key recommendations/ considerations from the last assessment note below and how these were actioned. 
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Key recommendations/ 
considerations from the last 
DFA (list each recommendation 
below, add more rows as needed 
General ODS DFA Feedback 
Focus on content design and 
information accuracy across 
all platforms including, web 
pages, webchat, and CIVR. 
Develop a plan to ensure 
content is being regularly and 
simultaneously updated on all 
platforms. Connect with ODS 
Content Chapter and attend 
drop-in-sessions to get 
guidance on content design. 


Kyle — Carolyn - Chuck 


Actions your team took in response to the key recommendations/ considerations 


1. Since the last DFA, Content has had a strong focus; taking an iterative design 
approach, from building conversational content, designing the questions with key phrases 


and responses, and tuning. 


: CIVR Building Conversational Content (GSiC) 
Iterative Desi SESS] 
rative sign Callers expect to be understood and get the right responses to their questions. 
Discovery Create Flows & Content Align & Test 
Anower; Whet are the most common simple Plan the experience along with content, The content, both questions and 


at poopie ask a 


IAT group to extabltsh common 


d. 
nossu 


fo sxamplenamy ca lers sateet 


‘the option far ~ ers ave Repeat fo the nextg cup ot top a 
tee. 


iey 
plates and wh der * Translate to Franch and upload to the 


bot, Test again 


CIVR Content - Questions and Responses i Tuning CIVR Content 


Questions — Tuning answers using the content designer 
; Ss Full Text Search - AWS CIVA system works to match a caller's question against 
the list of qoesti ers 2 ace. 


Ontario ¥ n Ontario '¥ 


The approvals process is supported the conversation review, the SWAT process with 
internal SO teams, and web and social media with MPBSD Comms, MTO comms, MTO Drive 
Test, SO applications to ensure content algins and is updated in a timely way. This process 
was developed as part of the chatbot solution and has been running for 1yr 9 months. 
e Issues are captured, assigned, and resolved for new answers, backlogs, content and 
language, - in Kanban Microsoft Planner 
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e Operationally placed with the team that works communicating the messages 


internally and aligning externally 
e CIVR answers the public questions as part ServiceOntario Customer Care, guidance 


of, and ongoing review of the webpages is a team effort to ensure consistency 


Digital Chat Content Updates and Content Changes Process Flow 


Process to review and support quick decisions to make changes to the chat solutions content and /or webpage ae 
content. ServiceOntario CMI, MPBSD Comms, MTO Comms & Policy, ServiceOntario Retail, ServiceOntario Digital ep |r | 


Adoption Branch, DEEP 


Author: Kim Monastyrsky, BRM. 


E 
4 
£ 
is} 


n (CMI) -6, 6.1, 7, 


3.2. 
© pilot - As required - DEEP — BRM and Technical & Busi 
71,8,13 


7.18,10 


eneral Posts Fles Wiki Issues & Feedback ~ SWAT Notes 


Fanati 
Nevsubee Vebesoe RTD oe e00 


The ODS training has been taken by a couple of the team and we had the opportunity to 
discuss if there is specific advice for chat. 


e Same advice as for Webpages 
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Use style guide and Hemingway 


Content has been developed based on listening to 800 calls, user research, 
published content, and as an MVP and will be updated once live - based on use 
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General ODS DFA Feedback 

2. Develop conversational flow 
diagrams which help to visualize 
user interaction with the system to 


Carolyn - Tim 


decide next steps on the process. 


2. The flows are depicted in a PDF Submitted for this DFA 


‘Transfer from 
ECCS SO IVR 


Grey rectangular box (require customer input) 
- bargeable~ voice activated only unless otherwise specified 


audio file/TTS prompt) 


Our telephone servi 


Front End Message 


information 
ce number or date of birth). Le 


n as 
et's get 


Initial Prompt: ow can nels you? 
IniiatRepoir Reprompt: Sey whet you sre caling s391 


+ 


Wet is on Ontario Photo ra 
card? (DV_OPC.Info) 
(DocChained) - 
OPCAppiy. 


How do apply for an Ontario 
Photo card? (DV_OPC Apply) 
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General ODS DFA Feedback 

3. Conduct qualitative research 
to better understand user needs, 
motivations and frustrations. 

a. Keywords or phrases 

b. What success means to end 
users (connecting to an 
agent, or being added to the 
queue etc.) 

c. Users that are adversely 
impacted by the current 
process and how CIVR can 
address those cases 


Kyle & Tim 


Customer Experience Office - Online Experience 


Compili rat 


Frustrated 
Disappointed 


Not U cr-friendly 


Customer Frustrations 
61% of the complaints are related to renewing a vehicle 


sticker (Q11); 11% are related to Driver's Licences (Q2); 2% are 


related to registering a vehicle (Q3) and 12% are related to 
other issues 
+ 54% of complaints are technical errors experienced by a 
customer when trying to renew a sticker which results in 
them being double-charged 
+ 21% of complaints are about users experiencing issues 
entering their addresses when completing a transaction 


+ 14% of complaints are related to the limitations of payment 


“Mone: 46 compal 


options (e.g. include debit option: more financial institutions) 


IVR Survey Sum 


“i voice 
agent” is adequate forthe content scripts. 
CIVR prototype /MVP. 
There may be an 
Insights from the MVP to improve 
would guide the design of (or create additional) 
a larger more complete scripts as users engage 
system. with the Conversational 
IVR prototype / MVP. 


Interview Results 


GovTechON 


* Review of the Customer Experience Office (CEO) 
complaints for the online services* indicate that users are 
frustrated with the online experience 


* Analysis of the nature of complaints show that an 
overwhelming number of users experience technical 
issues when completing a transaction online 


O 


Ao time period Apert 2010 10 Ail 


W200 At 


dear direction should be 
retested with wider 
audience. 


It is possible to include 
some of these types of 
questions in the 
prototype / MVP. 


“ Ontario 


We listened to over 800 calls 


People with disabilities 


How do I renew my 
photo carg? 


New drivers, >80 years 
old 
. Resta eponir fae 


+ Whee dot goto gets 
Saro lanea? 


+ Hes aver 30 nawe 
whatia the procedure 


3. Research has been conducted helping to create a mental model 
and has truly helped the team better understand user needs, motivations and 
frustrations. 


(GSiC) 


Uconced drivers Vehicle owners, We learned: 
. + ibai ti late 
wom i ani neta sorene 28 % of calls required look-ups by agents 


+ peed to replace my and cannot be self-served, 


1 wore pay fines a 
oan 


pane a + Lwes torepiace ny Top questions: 
| received a renewal permet 
nonce. Waat do | do? a benat to rapface oy * Delivery issues 
+ Wow do get ry ateher 
mapa ries Lenten re E -Online transactions (eg. Payment 
maa carto a toriy member issues, errors, didn’t receive 
confirmation) 
herna rem ag per learn E + DL history/abstract 
-ounie aren + Appointment booking 
ae pee RE User survey indicated: 
Las paa- h ce 
Sand into mah E dyn nage Respondents want a friendly, nurturing. 
iia . Pm visting Ontgtiet knowledgeable virtual agent where 
BERAY LAMM | Gan Idrive hose? there is open conversation with single 
need to 


audio prompts 


GovTechON Ontario ¥ 


After receiving 41 survey responses to the Microsoft Forms survey, the following high 
level insights have been identified. 


Themes provide are based on the survey data. 


1 Information provided (via phone agent) should be delivered in a friendly tone, with the 
opportunity to followup with additional questions 


2 Asingle response using existing the (existing) IVR/Web content was preferred over 
creating a new multi-prompt responses 


3 There was no dear preference in the personality of a phone support voice entity; the 
least preferred option was one that sounded direct and formal 


4 Multiple write-in responses contrasted that call messages were generally too long, but 
could be considered a reference to the IVR experience in total. 


conminemiaL—roncescimsion n AON Eee HON 47 Ontario @ 
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Survey- #1358 IVR users touchtone responses 


Servey Findings 
Conducted oe the DVPI phones from Féteas 
March &* 
- Dedicated survey for the IVR with 3 question 
- Stk Surveys offered with a normal response ra 
of 23% 
- For questions related to the IVR 
+ 1358 responses to question satisfaction 
+ 1329 responses to questice ease 
+ 1087 responses to question 13 (verbal) 
991 weld 
- Note the responses from callers were generali 
positive using touchtone feedback method 
- Top two box positive results 

+ T241% -how satisfied were you overall with 
the automated system before you were 
ransferved to an agent? 

+ 1248 %- how easy was it for you to fied the 
information you were looking for from the 
automated system before being transferred 
an agent? 

| BK Of the verbal responses were about the 


‘Satisfaction with the IVR system 


GovTechON 


Contact Centre Survey Analysis -Calls 


= The following insights from the survey may tell us how webchat can help resolve some of these issues 


Other Top Questions Received 


Of the other top questions received 


sem ante (GSC) 


Conducted on the DVP phowes from Fe 
16 two March & 


On a seale of 1toS where 1 means 
stromgly disagree and S means strong! 
agree how easy was it for you to find t 
information you were looking for from 
the automated system before Being 
transferred to an agent? 


Waar's the most important reason for 
Your score to the previous question? N 
you will hawe an opportunity te leave a 
recorded message up to 60 seconds lo 
After the tone please speak dearlyan 
press pound when you are finished. A 
reminder please éo not indude any 

persceal Information that would idem 
you or someone else in your feedback 
Any isfoematioe provided will be used 
anceymously for analysis purposes on 
and will not result in a call back to yot 


is there anything relevant to your 


experience calling into è goverment 
contact cone? 


Ontarlo Y 


32% are related to policy procedures that maybe found on the weltfiow do / get out of 
suspension? Mow do I create o RIN?” 
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Rewsits 
While 72% satisfied with ease of fieding information 


The WR did nct fare as well as we learned from the 
$32 valid verbal response 
+ 46% of responses were about the IVR 
*oweral! of the 991 stout the IVA- 12% were 
poshive and 24% wos negative with another 8% 
eed 
+ 24% of responses were about the apent 
overall of the 9591 about the agent ~ 17% ware 


poshte and 5% was negative with another 2N trsights 


We reviewed #991 IVR users Verbal Responses 


= li 


miad + Generaty people interactions are seen more favorably than techrelogy 


Themes 
General desatistaction with the broadcast 
message length kord related?) 

+ Vesd with multiple broadcast messages before 
reaching main menu 

+ Multiple ush of information 

+ Option not available |o.g, what to do F damaged 
sticker, lost Seense, expired license, booking 
aopanbrent, cance personalized licence plate) 

+ Zero. out option not clearly specifies 

+ Hoaring impaired = hard to understand 


+ In ys cave agents can offer more personal information 

© Since this wil remain the cane with CIVR we expect to see this trend to continue 

+ Theres supacet CMR concept and with some attention to broadest mmaages reqsired 
future merowements 

* CIUR decreases She number of leveti/levers. providing quicker access to imtormation 
+ Provides more and specific information answering more of the public's questions 

+ Voice wil humanize the experiance and speak in simpler language 

Expected results 

* More infeemed calers when speaking toan agent = shorter calls 

* More self-serve callers = less col to agent? 

+ Less negate feecboct about the VR overall — Improve overall CIAT 


+ More information 24/7; faster eqperence 


GovTechON Ontario Y 


Contact Centre Survey Analysis —Calls (cont’d) 


+ The following insights from the survey may tell us how webchat can help resolve some of these issues 


(OF the feedback received for Ontario.ca website 


27% mention it is difficult to navigate tebsitenr it is gi Mcult to find informa thor 
cant find information | need. The information à vague ond / m not sure | understand.” 


Zen minted ONDD- Vaat batas enn ea? Wet ere te ace hea? Wit fo? 
db if driver 3 ficence & exp ring? What are the extension times 


14% mention experiencing techmical issues or geting an error messà get an error 
message upoe renewing driver's licence or valtag adéeess online.” 


118 mention information is wrong or outdated 


23% are related to caller's specific fle or situatidang. what i the status of my driver s 
cence? Why 3 my licence sespended? Has my product been mailed?” 


Tæ Compan rts 
IEX are related to long wait times to speak to an agent 
20M ave delays in tansactions or mecahdng produ 8i days hove passed and | hove not 


‘rece ved my sticker renewal. When is it arriving? Why can t check the status of my 
(fie/product onl ne?” 


RO AO cf mporn we o ada Nana duets nom mpenta 


‘When asked about potential selutions to improve ca ler experience 


22% suggest mew process ImprovemantiChange IVI so that callers get an 
update on wait imes introduce o tichet number instead of looking by caller 3 
fame greater transparency to clients on timing af products” 


20% suggest improwements to the Ontario.ca website chat function for 
Layents to answer quick questions mare communication an COVID-19 updates 


Petal tokens o me ornate lage ace 
swo me 

ars mm oe 

a~ meee 

ah satine mm 


Be mo mme eme 


18% are categorized as OthefWhy can t / abeain my abstract or test results? Why wasn t 


(notifies? 


CIVR User Research- Callers prefer the CIVR over the IVR (GSiC) 


Recognized that if hey had general ques tos] Vibwould be helpful and 
seceessful in providing a quick aniwer knowing they auahways connect toa 
ramon agem f seeded. 


Anticipated that thesystemwiliprovide answers to their questions but 
then it would direct them to online resources or a human agent th 
provide more information on thelr question. 


When the participa spoke keywords clearhend did not have background 


Boise mest of the time the OVR was able to answer the question on thd to an actual human ASAP and aot need to aavigatahrough a taditional 


When calling most participants statedaipuld ke to be abie to talk 
m. 


thaQIVR systems were easier to aavigate for those who had IVR 
ee moat h ndgative experience with IVR systems 


OVR was preferred over VR OVRebminates the med to navigate 
joven aapisen alenke hornan tei 


With sQVR theycould just speak out the teywordirela ted EEEE, Agree d tha the intial information provided for general questions was 
heipu. 


Jenaugh and the amount of informa tion given would sutfid ently 
help them with next steps for thelr msue 


Keywords like “cost off accessible /disability/handicap parking permit” w The keywords “accessibility” & handicap parking pweré shecesstel 


racons in peting an armwer. 


in petting an arewer. 


Keywords like “selling vehide/ca r successful in getting an answer. 


Keywords lika “driver's linse suspended” or “driver's licaase ex 
wee seccensful in getting an armwer. 


GovTechON Ontario 


a. evwords or phrases 
Success was found with in testing and with research with AWS - it understands 
callers better by using natural language processing, artificial intelligence (Al), and 


machine learnine 


to understand callers and 


re ated to cesures office hows processing times” 


Mom JR acponcat we e othe fone due t non apc 


Dana 


provide them with the information the 
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need quickly with confidence. Example - understood “accessible” parking pass when a 
participant said “handicapped’. 


. What success means to end users (connecting to an agent, or being added to 


the queue etc.) 

Success is both the ability to self-serve for information and reach and agent if 
that is what people want. The MVP provides a far improved access for finding 
information on the IVR — confirmed in research. We know from data that XX% are 


listening to the information in the IVR already — information is a 24/7 service. 
. Users that are adversely impacted by the current process and how CIVR can 
address those cases. 


e Too much thinking listening and remembering currently ~TO~ Asking a 
question and being understood with a variety of ways to ask. 
Navigating many menus, multiple levels, calling again, zero-out ~TO~ Being 
quickly connected to the answer just by asking for information people want. 
Callers using the back button appearing trapped in a maze or continuous loop 
~TO~ more conversational and will understand what people are asking 
through good programming and Keyword use. 
Zero out option to agent is not clearly specified ~TO~ CIVR is designed to 
direct people to the queue for an agent by saying — “agent, representative, 


Understand users and their needs 


CSR...” 


Question [Response > 


ODS DFA Feedback 


e Assessors understand that the team has large amounts of existing user data to analyse as 
part of user research, however, that should not replace conducting direct end-user research 
Conduct active qualitative research, including card-sorting activities to validate keyword 
choices and iterations, This will also help the team understand what success means to the 
end users, what are their expectations from the CIVR and identify any extreme cases of users 
impacted by the current process. 

è Analyze the difficulty some users might have in navigating the menu in the current workflow. 

) i resear' zins on for help with your user research plan, induding 
recruitment and interview questions. 


How have you tested your MVP with 
real users? How many users have been 
involved in testing? How often do you 


e This is done through SWAT and gathered Key words from CIVR and the point 
is that there is a learning curve and the learning engine that AWS has that is 
much wider than the local information 


e We have partnered with ODS UX team to test the service from end-to- 
end: connecting to the IVR, then being handed over to a live agent if required 
or asked for. 


e From these user research sessions, we planned to: 
e Understand user’s thoughts and expectations of the D&V CIVR 
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conduct usability testing? 


Kyle 


e What are users’ expectations for the service? 
e What are users’ overall thoughts on the value of this service? 
e Would they trust this service? 
e Evaluate the usability and comprehensibility of the D&V CIVR 
e How do the users navigate the D&V CIVR? 
Does the flow and terminologies match users’ mental model? 
What would users do when they encounter an error? 
Can they find the help function/ do people use it? How would they use it? 
Do users understand what to do after they’ve used the D&V CIVR? 
Can users get their questions answered without speaking to an agent? 


e Usability testing has been done, by both cluster and operational content 
design - DEEP testers and 3 cycles with Service Ontario over September and 
October. 

e Further ODS user research with the CIVR and past experience with the 
webchat. 


Who are the different user groups you 
tested with and what were their diverse 
needs? 


Kyle and Matt and Jerome 


Internal user groups 
Email Survey — DEEP practice and family & friends (41 responses). Analyzing 
this data was to confirm how / where we could improve the existing IVR content 
within the CIVR Prototype / MVP system. Demographic details collected from the 
survey: 

- 33 / 41 respondents hold an Ontario driver's license 

- 6/41 respondents listed English as NOT being their 1st language 

- 31/41 respondents are between the ages of 30 — 60 years of age 

- 6/41 respondents are over the age of 60. 


Listen to 800+ callers and IVR Survey — 1400 D & V callers — representative of 
average Ontario callers 


UAT DEEP — 472 responses for testing — internal OPS users 


UAT ServiceOntario - 3 cycles 
Cycle 2 — 15-20 
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5 bilingual test both French and English 


User research groups diverse needs included people who might use to people 
who would never use and typically have support from family for technology 


Methodology 


About the Participants aca 


e Participants are from various cities across Ontario, ; 
encompassing both Northern and Southern Ontario. 3 
e Most participants are between age 30 to 45 and are 


familiar with Service Ontario. 


There is an even distribution of men and women. 
No participants reported any disabilities. 


6 


Participants’ Familiarity with SO 


V 


Gender Identity 


3 


Ontario @ 


Methodology 


About the Participants Participants Ethnicity 
e All participants indicated that they have 


1 1 «Eas Ast 

i 1 + Middle Easter 
immediate access to the internet most of aN y^ - 
the time. Saita 
Almost all participants indicated that they 5 


have the skills and confidence needed for 
most digital tasks. With only one participant 
indicating that though they can do most, 
they would prefer help. 


Almost all participants report speaking 2 
English at home, with one report speaking 

Spanish, and another speaking Urdu. 

Participants come from a range of ethnic i - 5 
backgrounds, and most participants have 

spent a significant amount of time in 
Canada. 


4 


Participants’ Length in Canada 


Ontario @ 
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What are some of the changes 
identified as a result of researching with 
users? What user needs are most 


This research looked a learning from real users for design and content 
Insights from caller surveys #1400, Listening to 800# Calls 
e 28 % of calls required look-ups by agents and cannot be self-served. 


Top questions: 
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difficult to meet? e Delivery issues 
e Online transactions (eg. Payment issues, errors, didn’t receive 
confirmation) 
e DL history/abstract 
e Appointment booking 
e While 72% satisfied with ease of finding information — 1358 responses 
e Verbal responses - The IVR did not fare as well as we learned from the 991 
valid 
e 46% of responses were about the IVR- 12% were positive and 24 % 
was negative with another 8% mixed 
e 24% of responses were about the agent - 17% were positive and 5 % 
was negative with another 2% mixed 
Themes 
e General dissatisfaction 
e Broadcast message length (covid related?) 
e Multiple broadcast messages before reaching main menu 
e Multiple levels of information 
e Option not available (e.g., what to do if damaged sticker, lost license, 
expired license, booking appointment, cancel personalized license plate) 
e Zero out option not clearly specified 
e Hearing impaired — might be hard to understand (there is TTY) 


The most recent research 
e We have partnered with ODS to test the service from end-to-end: dialing 
in to IVR and connecting with the CIVR - then being handed over to the live 
agent Queue 
e From our sessions, we confirmed: 

e Participants prefer CIVR over IVR. 

e Anticipated that the system will provide answers to their questions, 
but then it would direct them to online resources or a human agent that will 
provide more information on their question. 

e Agreed that the initial information provided for general questions was 
helpful enough, and the amount of information given would sufficiently 
help them with next steps for their issue. 
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Influence on design and changes: 


e Reported that CIVR systems were easier to navigate — for those who 
had IVR experience, most had negative experience with IVR systems. 

Agent Handover: 

e Recognized that if they had general questions, the CIVR would be helpful 
and successful in providing a quick answer, knowing they can always 
connect to a support agent if needed. 


e CIVR design will eliminate multiple levels of information and people can 
request and agent just by asking 

e Content added to CIVR based on information heard on calls and 
information identified as missing, and what the chatbot currently answers 

e In pilot system will understand keyword questions based on AWS Al — 
track utterances for any new keyword questions 

e Will continue to observe caller behaviour in pilot to match observed 
behaviour in research; i.e. asked longer questions until they learned to 
emphasize key words, or repeated CIVR phrases back. 

e Test if - upfront messaging changes can generate; fewer, simpler words 
and one question at a time in callers 

e For personal information/status - directing to status for Driver’s Licence 
tool, and online information with Suspension ID#, currently no tool for 
Licence plate or card delivery 

e The Contact Centre project team — review the broadcast message to 
simplify and shorten — measure and test reaction 

e Continue to survey to understand improvements and gain feedback 

e More testing and work on cost and fees 

e Backlogged ideas for further research 

e Add “previous” or “next” option for each paragraph for users to 
navigate through answers at their own pace. 
e Add a pause feature and “how to pause” instruction message option 


What common user flows were tested 
with users? 
Kyle & Kim 


End-to-end: using the CIVR testing environment to listen and ask questions with 
the Al chat to assess the experience of being transferred to a live agent from the bot 


3 scenarios using the testing environment with a dial in and move to queue — 
no live agent 
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e Scenario 1 — Suspended licence - “Your license has been suspended, 
and you want to figure out what to do next.” 

e Scenario 2 — Selling a used car - “You're selling your car and want to 
know more information about the process. Such as what type of 
information is needed. You also want to know how much it costs.” 

e Scenario 3 — Accessible parking permit - “You want to get a parking 
permit for your parents who have accessibility needs. You’re not sure 
how to get a permit and want to find out more. You’re also curious if there 
is an associated cost.” 


What data are you collecting to gauge 
the performance of the solution? Have 
you noticed any trends based on this 
data? 


e Currently a baseline and a targets have been developed to measure success. 
e See presentation attached on reporting 
e With the rollout the team looks at trends within 2 months of go live. 


(GSiC) 


System Monitoring — Word Cloud and Missed Utterances 


Learn about the what people are asking most — Voice of the Customer. 


Reporting — Starting with similar reports, and word cloud for focus. Over time the data will be enriched; contact 
handling quantitative statistics will include speech analytics capabilities (for clients that use the EP-cloud instance) 


DV_PV.RenewSticker 
DV_DL.DriveTest °or Pm ucence Rene 
V.Permit Replace Dy _DL.DigitalReminder = ” 


si empty 


-DV_PV. Transfer 
DV APP St 
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Tra 


System Measures — Trace Report 


We can trace what people are asking about. 


ce Reporting 

This is a view is available for a 
requested time period 

Each conversation is assigned 
acode 

Time stamp shows when the 
calls came in (UTC) 

The conversation is outlined 
by the naming conventions 
for each item 

Time is noted in number of 
seconds for each named 
convention 


System Measures — Queue ID - Navigation 


Each a topic/queue and is identified under question ID. 


Queue ID Reporting 


This is a view is available for a 
requested time period 

Begin shows the total number 
of conversations started 

Each topic shows how many 
times this was used, it will 
always add to more than the 
beginning because more than 
one topic can/will be included 
in a conversation 

Dates represent the volume 
by months as this is selected 
Columns are similar to 
current IVR navigation reports 
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Establish the right team 


Response 


ODS DFA Feedback For this next phase ServiceOntario was a contributor and has now added resources 


e Review roles and tisporditáiies ol uate members to ensure there are no gaps, and that to the core team 
ere cnt peor eee rn ro e ServiceOntario Sponsor Sr. Manager is Beth Preszcator of Change Management 
and Implementation, Customer Care with 3 roles on reporting on the core team 
e ServiceOntario Product Manager 
e 2 team members for chat content design, and organizational processes 
ServiceOntario Sr. Manager is Lisa Boffa Contact Centre Modernization and 
Implementation, Customer Care 
2 people for organizational readiness 
Does your team continue to be The core team includes GSIC, in collaboration with ServiceOntario CMI and 
multidisciplinary? Share details of the Contact centre, with broader project team, partner Services, communications, 
: policy, ODS research, measurement and analysis, MTO policy and partner 
current expertise and roles on your services. 
team. The product leadership for the pilot is shared with the technical build lead by 
Tim Trudeau DEEP and Operational and Content build lead by Jerome Song 


ane O - Worked on the team of ServiceOntario Customer Care — Charles Lavigne 
in the pas 


Driver and Vehicle Conversational IVR 
Technology creation was led by DEEP 

e Project/Product Manager — Tim Trudeau 
1&IT Lead Technology Specialist - Annie Perez 
Service Designer/l&IT Lead Technology Specialist - Carolyn Luney 
1&IT Lead Technology Specialist - Charith Senaratne 
1&IT Technology Specialist Co-op — Diwakar Singh 
Content Design/Accessibility - Kyle Kwan 
Content Designer/User Research - Lyn Lim 
Service Designer - Julie Buelow 
Service Designer & Project support — Kim Monastyrskyj 
Business Relationship Manager — Sharon Diaz 
Sr. Data Analyst/ User Researcher - Anitha Robert 
Business Analyst - Collin Brown 
QA Specialist - Matthew Ha 
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QA Analyst — Gleb Khoroujik 
QA Tester - Julia Galeotalanza 
QA Tester — Esha Sarfraz 

QA Tester — Parneet Lubana 


The core team from ServiceOntario are from CMI — leading the plan for the day to 
day for readiness, ongoing process for building the conversational content and approval, 
with the support, guidance from DEEP 


Lead Sponsor Manager Beth Preszcator 
Product Manager Jerome Song, 
Content Designer Charles Lavigne 
Content support Nancy Lajeunesse 
Contact Centre Malaika Christie 
Contact Centre Neal Mendes 


Executive 


Executive Sponsor - Nelson Loureiro, ADM, Customer Care, SO 
Governance - Manish Agarwal, ClO, GSIC 

Product Sponsor - Olga Bakonyi, Head, DEEP, GSIC 

Director Contact Centre, Customer Care - Mary Ben-Hammoud 


MTO SMEs 


Sr. Policy Advisor, Driver Program Development - Keith Madill 

Sr. Policy Advisor, Driver Program Development - Arthur Legardo 

Policy Team Lead, Vehicle Program Development - Bob Rajhan 

Sr. Policy Advisor, Vehicle Program Development - Andrew O’Connor 
Manager, Program Standards, Licensing, Permits and Support Office - Melody 
McClelland 

Manager, Licensing, Permits and Support Office - Jennifer Elliott 

Team Leader, Contract & Oversight Management Office - Julia Scorcia 
Business Advisor, Contract & Oversight Management Office - Maryanne Cirillo 
Service Designer — Digital Garage - Vergel Evans 


MPBSD & ServiceOntario SMEs 
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Communications 

e Praveen Senthinathan, Ashleigh Cocuzzi and Derek Kinley 
Contact Centre Services 

e Alessia Vaughan, Chuck Jane, Rover Okumura, Lisa Boffa 
Change Management Implementation 

e Sarah Loveridge and Trish Lewis 

Reporting & Information Management - Carine Chartier 
Privacy - Peter-Hope Tindall, Head Privacy MPBSD 
Business Training Unit - Stephanie Guillen 


ODS SMEs - have been engaged for research 


Senior Service Design Specialist - Dana Patton 
Senior UX Designer - MJ Singh 
Experience Design Co-op - Emily Yan 


The project balanced what can be delivered with the knowledge, time and funding 
we have, working with the support of our technical vendors and internal 
developers, testers, stakeholders and research — to deliver an MVP to pilot. 
Decisions were based on the developed web chat framework, guidance from 
ODS, response from the public and team input 

There has been ongoing research and partnership with DEEP and ODS to 
determine what the public expects and needs. 

Leadership and Operational Management for this digital solution, including 
content development and approvals, is housed in the sponsoring ADM’s area in 
ServiceOntario, and will have pilot partnership support from DEEP 

Worked with the enterprise design standard, technology and learning from other 
chat services to create the technology platform for CIVR on the enterprise platform 


Who is involved with building the 
service? Who is involved with testing 


DEEP has an enterprise responsibility for chat solutions to the public using 
the enterprise platform. 

This CIVR is in partnership with the business offering the service - ServiceOntario 
Currently there are IVRs, transactional IVRs, 
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the service? 


e DEEP has a testing team with an onsite lab, we have submitted the testing plan 
previously. 
e Currently Usability testing is being created for Live Chat and will involve all 
the partners involved 
e User Research completed May 2022 — has tested with 11 people from the 
public and we have been offered some advice to consider 


Tell us about your product manager or 
project lead's autonomy with regards 
to rescoping and changing the 
direction based on user feedback? 


DEEP Product Manager — Tim Trudeau 
Digital Enterprise Engagement Practice 
e Has autonomy and has worked with the team to scope and rescope as 
required to launch the MVP — will continue in partnership with ServiceOntario 
e Sits on regular planning and project team meetings to inform them of how the team 
is proceeding — IT implementation planning 
e DEEP is refreshing along with the Ministry and GSIC on how we approach our 
work as a practice 
e DEEP has: 
e Recently hired developers and data professionals to fill in gaps 
e ls training on Human Centred Design and Lean 
e Establishing its strategy and approach to product management 
e GSIC PM training 


ServiceOntario Customer Care Product Manager — Jerome Song 
e Working in collaboration with DEEP — leading ServiceOntario: 
e project process and change management 
Autonomy with 
ensuring content approvals process and approvals are in place for rapid change 
operational readiness, 
UAT testing planning and conduct 
Communications and MO updates 
Planning show and tells 
Reporting and Measurement with RIM and BI 
This digital product is new and may be repositioned in customer care with new 
director hiring 
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As a Ministry - word from DM of MPBSD - Renu Kulendran September 19th 

“While we know we’ve sharpened our focus on digital service delivery 

excellence, we're still finalizing what this means for broader organizational 
realignment. We should have those details within the next few weeks and, as 

our organization continues to evolve, we will be sure to communicate regularly with 
you about what the changes are and what they mean for you.” 


Who is the executive sponsor? How do 
you brief them, e.g., show and tells, 
briefing notes, presentation? 


ADM of Customer Care ADM Nelson Loureiro. 


DEEP held several show and tells to MTO & ServiceOntario partnership 
meetings — GOC (October, April, June) 

Leadership meetings with directors and ADM - regular 

Planning meetings with Customer Care leadership for prioritizing projects 
Project meetings and workshops held with representatives from policy, privacy, 
TRA and MTO and MPBSD 

Regular Weekly core team updates 

Show and tells for ServiceOntario leadership, DEEP Practice and All Staff @ 
GSIC 
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Be consistent 


Question i Response 


What the team needs to explore going forward 


+ Knowing the inconsistency that already exists in the system, the team is encouraged to look SWAT as outlined in several answers has worked to improve the 
into setting up periodic touch points between different SO teams to maintain consistency as consistency with regular meetings, Kanban f and collaboration on content 


a day to day operational item rather than one time exercise. 


It is understandable that with the implementation of CIVR, more information could be To start with this takes much collaboration a There will be regular 


provided to the user. As the gatekeeper of information, CCMB needs to ensure that the 


information provided to the user is strategic and only essential. meetings, reporting will be used to watch the flow of calls and users 


The product owner and product manager from CCMB should hold the pen on what 


goes inside the CIVR for the public consumption. The ministry partner's involvement DEEP will manage the input in this MVP as they continue to work with the 


should be on the information accuracy basis whereas the development of content 
would still be done internally by the product team through their in-house content y: t m 
specialist S S e 


Attend Contant Dasien drop-in sessions to get guidance on content design ServiceOntario approves the content and works with DEEP to implement 


Demonstrate how the service has used DEEP is the provider of enterprise voice and chat solutions ensuring 
the Ontario design standards. Does your consistency of applications , , 
2 > DEEP (CCMB) has been working with the ODS for over 2 years 
MVP use the Design System: developing chat solutions framework and influencing the enterprise 
Carolyn design standard for all Chat 

There is no change to how the phone number is published on the 
ServiceOntario Driver and Vehicle webpage - “Contact Us” pages on 
Ontario.ca. 
DEEP is the design authority for the Engagement Platform, including 
voice applications - standard messages and language are used from 
design standards where possible 
We expect there to be ongoing learning with this technology upgrade and 
we will continue to bring the understanding of caller needs as a priority 


Current examples used: 


e Repair paths 
e Voice applications 


Does your service use Ontario.ca as the e Ontario.ca publishes the phone number for in “Contact Us”. 


main entry point? 
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Show how you’ve applied Ontario.ca 
content standards to maintain the tone 
and voice of government content. 


e The scope of this project has been limited to Driver and Vehicle Services, 
using published information from webpages, social media, News Releases, 


and how contact centre agents reply 


e All these sources follow the content standards for Ontario.ca 
e Response information has been adjusted as per the content design 


requirement of the AWS CIVR 


CIVR Content - Questions and Responses 
Responses Capture 


Questions 


(GSiC) 


* Questions are in the form of a sentence 


* Not one-word only 
* Not only Keywords 


* Be specific with the question: 


+ How do I renew my driver's licence? 
+ How do I change my address online? 
* What are the requirements for a photo card? 


* Format correctly to ensure it is properly 


imported to the CIVR, or it may generate 

errors. 

» Use punctuation at the end of the question or 
sentence — even fragments. 

* Do not use slashes - written out as you intend it 


* Keep responses short (30 sec) and relevant - offer the most 
important information first. Caller needs before business. 


Inform callers - how they can respond or what actions they can 

take. 

* Callers can interrupt or ask for help at any time — response 
should reorient them to their current context within the 
application. 

* Confirm the topic area — ‘echo back’ 

» “Ok, booking a renewal session.” 
* Topics can share the same “echo back” phrase 
* Simple responses: 
* “Before your 80th birthday, you will receive a renewal application form 


Document: 

Question ID, with 
correct format. 
DV_PV.Permit_Replace 
Cost 

“Topic: 
DV_PV.Permit_Replace 
-Conversation Model: A 


Questions: the 
utterances, questions 
that the user will say to 
return the response. 


* Response that is a reasonable length 
(about 30 seconds), 

* 90% of the answers found should be 
simple. 

* Cannot barge-in during first sentence of 
the response — same if only sentence. 

- Adjusts for the caller’s environment - 
noisy restaurant vs a quiet house. 

* Create a natural transition using words 
like “okay” and “alright” and “sure”. 


when there is a related piece of info 
that callers may want to hear. answer. 

- Use the sparingly. 

* If used too much it could create a 

confusing/repetitive user experience. another model C. 
= Questions that aren’t too closely related 
to each other (plate sticker, renew DL) after the answer 
* Offer an idea of what they might say 


next. 

* Caller is returned to the “main 
menu” after the answer, where they 
are prompted again (e.g., You can ask 
me something else) 

* Caller can answer guided prompt or or 
they can ask something else. 


question before proceeding to the 


* When using this model, the response 
should be is linked model A or B - not 


* Caller is returned to the “main menu” 


= In development stage. 


* Model C’s use a mini lex bot and requires a 
developer, in addition to the base 


* Caller enters a model C, they are expected 
to say one of the accepted utterances (e.g., 
online or in-person). 

* By default, they can also say agent or start 
over to exit the model C. 


iibe pibgiarned (slasties are OK in responses): and a letter in the mail and you will need to attend a renewal A AA 
+ URL can be useful in an answer — no dashes and SESSION. À 
periods signal it is ok to barge — use “dot”. * Guiding follow-up response 
+ “What is the cost” question applies to many + “If you want to learn about what to bring to a renewal session, just 
topics may need an ‘echo back’ when tuning. ask” 
* End each of the CIVR’s turns with a question, prompt or Call to Action. 

6 oviechO Ontario Y 
CIVR Response Content Models ( ) 
Model A Model B Model C — beyond MVP 
- Use Model A for simple answers. - Use model B for guiding follow-up - » Use model C to ask a clarifying 


27 Presentation Name 
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Tuning CIVR Content 


Tuning answers using the content designer 


Full Text Search - AWS CIVR system works to match a caller’s question against 
the list of questions and answers stored in Amazon OpenSearch Service. 


Aconfidence score - is created for the best match for the question asked from 
the full text search. 


« Words that are used infrequently score higher than words that are used 
often 


-Keywords filters are applied, used are matched to the question with 
preference to the same words, used in the same order. 


«Sentence constructs such as prepositions have lower weighting than 
unique keywords. 


Confidence at or over threshold set - has the greater the probability that the 
best match is the most relevant and it will be chosen. 


Callers may ask questions in ways that you haven’t anticipated. 
* May resulting in unexpected answers being returned. 


- When this happens, you can use the content designer to troubleshoot and 
fix the problem. 


More AWS information and tools 


Tuning Testing Troubleshootin, 


Tuning Automatic SR 

Testing Questions 

Monitoring usage and feedback 
Tuning Accuracy 

AWS Glossary 
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Show how you’ve created a consistent 
experience across all channels. 


e CIVR answers the public questions as part ServiceOntario Customer Care, 
guidance of, and ongoing review of the webpages is a team effort to 


ensure consistency 


e The development of an enterprise framework for Chat solutions done in 


partnership with the ODS 


e All Public Chat Solutions are to be delivered through DEEPs Engagement 


Platform, and use the design standards 


e Created to ensure consistency across all channels and partners 

e The process for issues and content updates was established and used 
for the pilot with chatbot — updates made for both chat products 

e Improvements have been made with ease of change — 


ServiceOntario manages content as follows: 


Operational Content 
e Operational - Published content is approved and changes of content for 
chat considered operational and no comms or policy approvals required — 


JP Director Comms (Sept 2022) 


1. Content as is (translation only for intents, questions or buttons) 
2. Plain language content 
3. Adjust for readability level and flow (Hemingway) required as per ODS 
4. Standard conversational messaging designed in Chat Framework 
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5. Content removal — not aligned, tech issues 


New Content or Issues 
e New content MPBSD Comms - web, chat, applications — 
o require a form with analysis, and proposed content — comms 

handles approvals (may share translations) 
includes suggestions based on “voice of the customer” 
goal for the public to find the most common answers on the web 
and in the application first 

e Errors found are managed urgently through web coordinator and 

changed rapidly — CAB/MO for some 
e Operational Content — FY! only 


SWAT for chat promotes the collaboration across channels and between, 
MTO, ServiceOntario, and MPBSD Communications, GSIC-DEEP to ensure the 
messaging is the same and updated simultaneously 


BD General 
7 Sk bh avreE  _ 


m a — ~ ao = —— 
pa 4 a a = am ç eoo: Sme 
Ee a aini ah d a a i - : 


e Issues are captured, assigned, and resolved for new answers, backlogs, 
content and language, 

e Operationally placed with the team that works communicating the 
messages internally and aligning externally 
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Design the service from start to finish 
(In this case ‘service’ means the full end-to-end experience for the user not just the system or IT portion of the service.) 


Question Response 


hat the team needs to explore going forward 


Have a lean practitioner review the current and proposed workflows for CIVR to ensure an The CIVR is a recommendation of a Lean Assessment from 2020 that include 
here E ae and Continuous improvement Office for guidance and request implementing Live Chat, Chatbot, and CIVR 
review the OPS Lean Playbook. 


support with lean deliverables. Consider having the team also rey { 

Assessors understand that the scope of the project is limited to only one step in the 

end-to-end user journey, however, the team can develop a roadmap of the product to i i i 

identify areas of improvement and address the overall customer experience which will IVR flows and navigational flows were reviewed . 

ultimately help in the adoption of CIVR. Diker and ikide co diboa (GSiC) 

What do the numbers say in 2020? 

Calls for FY2019/20 What has 4? 

Dialed - 172M call attempts were made to reach MTO services Published telephone numbers 

The overall customer experience of callers could be improved in multiple ways including: ~ 2023/20) © 85% Calis connected - There were 1.47M or $5% heardthe Welcome Pernera 
Looking at the volume of incoming calls for IVR and estimating what process flow is j Q eter service: nore pane 
impacting the bulk of customer experience =. implemented chatbot for D & V on 7 
Looking at the extremes or the adverse cases and identifying which subsection of a -r yo AR ene Leas trunk flow 18.5% 0222 
users are impacted the most teen, = vehides Increased agent staffing 

Team needs to clearly understand and define what success means to end users as well as 

the business. User research with the target audience, as detailed in Understand users and oe ~ hema ney A Oe eee 

their needs, may be needed to accomplish this. “| Se æ: determined (IVA navigational report from ECCS vendor CTT) 

© 15% Fast busy -0.25M (1.72-1.47) — There are 250K or 15% callers that 

are unable to get through as the queue cannot handle more callers they 

hear a fast busy signal. 


(GSiC) 


Of callers that reached the IVR- 12% are hanging up in the 
wR 


Delivery times 


GovTechON Ontario Y 


Ontario Y 


What is your plan for soliciting feedback | Soliciting Feedback 
e Plans are to repeat the survey tools used to set the baseline for 
customer satisfaction — in the IVR and after agent. 
for continued support of this service? e The plan is to do the survey 3 additional times to start 
e Immediately after implementation (to get initial feedback), 
e 3-month mark (this will give us insights into how well the new system 
has been accepted) and finally at the 
e 6-month mark (this will give us insights on how well the system has 
settled and also do some trend analysis). 
e Provided opportunities for the contact center agents to provide any feedback 


on an ongoing basis? What is your plan 
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e The team recognizes there are iterative opportunity for further versions 


beyond MVP 
e Continued support for this service is as with all enterprise services 
e Plans are for a staged operational implementation based on pilot results 
e DEEP continues to support with maintenance and support 
e CCOPs for issues 
e Business Relationship Managers provide support for new projects 
and are regular contact to support the success 


Explain how you have worked 
to create a seamless end-to- 
end user experience and how 
your user testing speaks to 
your success in this area. 


The end-to-end customer journey for an overall contact centre experience is 
illustrated in the figure below. The steps are: 
1. Problem to solve or information to find 


2. Research (may include website, social media, word of mouth, referral, etc) 
3. Call the number and enter the IVR 
4. Listen to IVR and make selections 
Our project resides in this stage of the journey 
The customer can exit the system at this stage through: 
i. Problem resolution through self-service (Go to #8) (ie information 
provided in the IVR) 
ii. Information provided in IVR to further customer on their journey. (Go 
to #1) (ie found information to connect to another organization) 
Press zero and/or wait in queue for an agent 
Speak to agent once an agent is available 
7. Call ends 
Customer can be transferred to team, tier (Go to #5), or another contact 
centre (Go to #4) 
8. Problem resolved or information found. 


D o 
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The scope of this project is focused on upgrading the IVR portion of the experience 
for the client. 
e People choose the phone and discover a self-serve option for the information they 
seek — they only need to ask a question. 
e Previously a limited number of topics were found by listening and selecting a 
number to reach information. 
e The information assumed most people want was available in a 5 by 5 structure. 


User research set out to test the service from end-to-end: connecting to the IVR, listening 
to information (or selecting and agent) by asking a question and given the information. 
They can ask for help or an agent anytime and after 3 tries they will be guided to the 
agent queue during contact centre hours. CIVR is available 24/7. 


People told us they: 


Start looking for information 
e Primarily with a Google search to find the information and sometimes a phone 
number 
e Some will call before they ever go into a ServiceOntario centre 
e In some small centres where they do not anticipate a line and know the people, 
they may drop in 


When calling this number they: 
e Will get the number from the ServiceOntario webpage or from the specific 
webpage “contact us”. 
e They assume there is a long wait-time 
e There is a route to get to an agent 
e They find the “broadcast message” long 


From the IVR (From the research sessions) 
e Anticipated that the system will provide answers to their questions, but then it 
would direct them to online resources or a human agent that will provide more 
information on their question. 
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e Agreed that the initial information provided for general questions was helpful 
enough, and the amount of information given would sufficiently help them with 
next steps for their issue. 

e Reported that CIVR systems were easier to navigate — for those who had IVR 
experience, most had negative experience with IVR systems. 

e Agent Handover: 

e Recognized that if they had general questions, the CIVR would be helpful and 
successful in providing a quick answer, knowing they can always connect toa 
support agent if needed. 


Success from research and testing 
e Participants prefer CIVR over IVR. 
e The system generally understood the callers 
e People learned or adapted to how to ask questions very quickly (2"? question) 
e They said they would use this system 


How have you prepared users e The perception of a long broadcast message is not part of the scope here. 
for parts of the journey that ° These messages are generally mandated from a strategic lens by Cabinet 
and Communications 


aren't directly incorporated e The CCMI team in customer care is aware of this and will review it. 


into your service? What do 


users need to know or do e Set expectations 
outside of the MVP to e Research confirms callers expect simple 
e Messaging reinforces privacy in the up front 
complete the task? e Offers of Help — to explain how to ask questions 
e Features 
e More information customized to what the public wants to know - Extensive 
work has been done to understand why people are calling and to include that 
information 
e Ask for an agent anytime and will be taken to the agent queue 
e There repair path to press zero to speak to an agent 
e After trying three times to ask a question, and are not understood or this 
answer does not exist, callers are taken to the agent queue 
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Directs callers to the best channel for a transaction or assistance — Online 
first, in-person for registering a vehicle, the agent for specific or complex 
information, location finder to information on In-person, appointment booking for in- 
person 
What are the policy or The CIVR is an upgrade to the IVR and guided by the caller’s voice - designed 
for general information with no need for the caller to explore webpages, or confirm 
. the information on the other channels. Most commonly requested information is 
been or could be made in there. This is designed to offer more information with easier access. 
order to create an end-to-end Result of the lean assessment to empower the caller with earlier and easier access 
service? How have you to the information they are seeking 
No policy changes required 


program changes that have 


actioned these? 


What are your communication e Communication advises there is no need for a snapshot or media Q&A 
It is a very soft launch with a percentage of calls directed through the CIVR to 


lans for the upcoming public 
P p 8P enable the natural flow to be observed and to ensure a manageable service levels 


beta launch? 


Ensure users succeed the first time 


Where in your MVP have you There is upfront messaging that explains the service that answers 
questions and can take you to an agent 
e “Our telephone service is a virtual assistant that can answer simple 
questions and connect you to a support agent when needed. This 
conversation may be monitored or recorded for quality assurance and 


included: 


Instructions, explaining 


who the service is for and training purposes. It is important you do not say sensitive information (such 
how it works? as your social insurance number or date of birth). Let’s get started!” 


Helpful tips? 


MPBSD 23-071 Record 10 MidBeta-Assessment Final-CIVR-2022-09-26 Page 30 of 52 


e Contact us or support e “Using your own words, you can ask me about driver, 
vehicle, and road information services. For example, you might say, “How 
do | renew my driver’s licence?’ Or” sticker replacement.” If you’re not 
e Error messages? Feel free sure what to do, just say, “I need help.” 
to share pictures through 


options? 


e “Using your own words, you can ask me about driver, vehicle, and road 
information services. For example, you might say, “How do | renew my 

cloud location. driver’s licence?’ Or...”sticker replacement.” If you're not sure what to do, 
just say, “I need help.” 

Help examples 

e You can say Help to learn how to navigate the system or ask another 
question.” 

e Using your own words, you can ask me about driver, vehicle, and road 
information services. For example, you might say, “How do | renew my 
driver’s licence?’ Or...”sticker replacement.” If you’re not sure what to do, 
just say, “I need help.” 

e “OK. Before we end our call, did | answer all of your questions?” 

e “Did I answer all of your questions? You can say Yes, No, or Start Over” 


an email or hyperlink to 


Callers can ask for 
e help anytime 

an agent at anytime 

the script to repeat 

repair path press zero for an agent 

trying three times to ask a question, and are not understood or this 

answer does not exist, callers are taken to the agent queue 

How are you capturing user There are reports to support ongoing improvements 

e Metrics - Provides data related to the usage of the CIVR. This includes 
what the CIVR understands the callers are asking, what answers are being 

complete the task they set out to provided, and the topics count for select period. 

e For example, this can provide insight into what specific information 
customers are looking for from one month to another which can 

acting on this data? be used to update webpages, and for continuous improvement. most 

frequently asked questions or missed utterances. 


interactions that could not 


complete online? How are you 
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e Trace and Missed Utterances — CIVR collect what it understands the 
callers’ actions are while using the CIVR which provides understanding 
into caller behaviour. Understanding how easy or difficult it is for the 
callers to navigate the CIVR — if there are many timeouts or max invalids 
that could indicate the CIVR does not understand what the caller is asking, 
and they are having difficulty. 

e For example, review the Trace report to see where callers are 
repeatedly asking CIVR to “repeat” the answers which could be an 
indication that the information presented is not clear or too much 
information is provided. As a result — specific script(s) should be 
reviewed and modified accordingly. 


e CIVR Navigational Report - CIVR initial version of a traditional ice IVR 
Navigational Report which provides a summary of total times a QuestionID 
was referenced, hang-ups, timeouts, repeats, requests for agents, etc. for 
each Question ID. It will also provide total number of callers and average 
time spent in the CIVR (displayed in seconds) 

e As an example the QID ServiceOntario Locations (DV.Locations_Info) 
from July 1st to Sept 15th was referenced 3 times and with 1 agent 
request. Analyzing the CIVR Nav report for this QID indicates that 
the callers are satisfied with the information provided due to the fact 
there are no invalids, timeouts or max repeats/errors. Generally, the 
callers are receiving the information they required and had no further 
questions. 
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How have you ensured an 
intuitive and concise workflow for 
users? 


System Monitoring — Word Cloud and Missed Utterances (GSiC) 


Learn about the what people are asking most — Voice of the Customer. 


Reporting — Starting with similar reports, and word cloud for focus. Over time the data will be enriched; contact 
handling quantitative statistics will include speech analytics capabilities (for clients that use the EP-cloud instance) 


Missed Utterances Count 
wanes = 
utterance Count s - 
rr | eee DV_PV.RenewSticker 
aonel need to pay 1 shiuih DV_DL.DriveTest DV_DL.DriversLicence Renew 
amom 1 lihava got my face N_PV.Permit_Replace DV_DL.DigitalReminder 
a me Umit to submit my aplatie to apy., 1 FAE mynte for the card eaei EMpty i 
2 wel need to apply for ender you for twent DV_DL.SuspensionsFines "Sy pv.registervehicle 
apard 
< DV_PV.Transfer 
leg main 


DV APP DV_PV.SpecialPermit 
= DV_DL RenewalSessic 


am i eligible for ana pp 


Total 1,194 


(GSiC) 


System Measures — Trace Report (GSiC) System Measures — Queue ID - Navigation 


We can trace what people are asking about. Each a topic/queue and is identified under question ID. 


Trace Reporting 
+ This is a view is available for a 


requested time period 
* Each conversation is assigned 
ac 


‘Queue ID Reporting 
hi avail 


hows when the > Each topi 

(uTC) times this was used, it will 
always add to more than the 
beginning because more than 
one topic can/will be included 
in a conversation 


i 
call 

* The conversation is outlined 
by the naming conventions 


for each item 
+ Time is noted in number of 
conds for each named * Dates represent the volume 
convention by months as this is selected 

+ Columns are sir 

current IVR navigation reports 


e Content and flow were based on user experience, workshops, and 
research 
e Recent research confirmed the CIVR preference over IVR and the caller's 
mental model 
e The team in now reviewing the findings and recommendations for short 
and long-term planning, and where further testing is required 
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Test the end-to-end service 


[Question ss Response 


How have you ensured the MVP is 
scalable? 


Tim and Carolyn 


Did you separate content, design and 
functionality so updates can be made 
independently of each other? 


What browsers and devices have you 
tested the MVP with? Please list all 
browsers, screen readers, mobile 


devices. 


How are you continuing to test with 
each new release? 


DEEP is has an enterprise offering and it is intentionally a 
scalable solution to engage the public 

The team has worked with vendor to ensure that it is scalable for 
all our clients 


Content, design and functionality can be independent streams 
of business that need to interact and learn together 

There is a core team of people working together to build these 
streams of work 

The technology and design is ready with the content approved 
There is design along the client journey to enable a content flow 
in the CIVR 

Content is developed to align to all other channels available as it 
needs to be consistent and intentional 

In CIVR the content is held in AWS cloud 


DEEP has dedicated lab and testing team for this work on all 
technology they produce — based on the ACOE standard 
e This is Voice not Web 
e This is not reliant on testing different phone systems or 
hardware on behalf of the caller 


Dedicated DEEP team tests all new changes and completes 
regression testing and application functionality 
Business testing is ongoing with the following testing 
Cycle 2 Scenarios 
1. A friend has reached out to you for assistance. They have 
been advised by their physician they will require a parking 
pass due to an injury. You are calling the IVR to find 
information on how to get this done. 
2. You have lost your driver's licence and need to know what 
documentation you need to have it replaced at a retail location 
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3. You are visiting a relative, you need to get to a ServiceOntario 
location to perform a transaction and would like to know how 
to find the nearest office. 

4. You are an international student and would like to apply for a 
driver's licence. What documents do you need? 

5. You have some fines and would like to pay them, however 
you're not sure if they are vehicle fines or driver fines. You 
want to find out the difference. 


e New content testing process with each update involves 
Content loaded 

Tool includes a testing tool - image 

Runs through the QIDs 

Identify conflicts the team works to address this based on if it 
is technology or content 


0.Global_ Handler. Agent 


Does the MVP have the ability to resume e This is non-transactional and information only service 


actransaction-atteran errar recover? e Inthe conversation there is the ongoing ability to ask 
y: questions 
Will users have to re-enter any data? e Callers can request to repeat 


e Ifthe bot cannot understand the question after 3 tries it will be 
sent to the agent queue 
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How do you track aborted transactions 
or flag potential blockers? 


What contingency plans are in place in 
case of a service outage? 


What the team needs to explore going forward 

e Include participants with accessibility needs who would be more likely to use the service in 
direct user research. 

e DFA team can connect the delivery team with the Canadian National Institute for the Blind 
(CNIB) to recruit users with accessibility needs for the purpose of user research. Please 
connect with the UX team to get support 

e Explore ways to ensure the CIVR options are easy to use for a diverse set of audience 
including people with language barriers. 


Did your user research and/or testing 


include users with accessibility needs? 


Reporting is explained in other responses 


If CIVR is not working it will be routed to an agent queue. 

Another key advantage of AWS over traditional web hosting is the 
Availability Zones that give you easy access to redundant 
deployment locations. 

Availability Zones are physically distinct locations that are 
engineered to be insulated from failures in other Availability Zones. 
We are using the Canadian availabili 


ODS has recommended testing for people with disabilities in future 
rounds. 

Testing was completed with people with ESL 

Inclusion checklist submitted June 2022 


The team reached out to the Ontario Accessibility Centre of 

Excellence for guidance on accessible practices for conversational 

IVR. 

Some user groups will benefit from the conversational IVR 

technology such as: 

e People with visual loss/weakening who don’t use or rely on 
visual or text-based information. 

e People with mobility or dexterity impairments who find buttons 
difficult to press. 

The team has ensured the CIVR Prototype / MVP system adheres 

to the same standards as the IVR system. 

The recruitment process does not always result in people sharing 

their accessibility needs — even with direct recruitment 
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e ODS has recommended further testing with the accessibility 


community in future research — they are working on this 
What improvements were made based on e Language was adjusted during content programming and have 
been adjusted based on testing 
e Simple language to help all callers understand easily - Moved 
“Purchase” to “buy” 
e Shortened content for ease of caller understanding - using 
the Hemmingway application 
e Programmed as “Accessibility” and people in testing and 
research say ‘Handicapped”, “Disabled”, “Wheelchair” 
e Usability testing indicated issues with type C questions — assist 
the caller when they are less specific with their question 
e The team has built mini “Lexbots” to support type C 
e Type C questions are available 


How have you incorporated accessibility e Yes see ACOE 
best practices into the MVP? 


user testing or testing in general? 


What other channels or supports are e People are directed to TTY, or web, social media and in person 


er b 
offered to meet accessibility needs? i ca 


Describe the manual and automated e Most testing is manual with an IVR using voice — outlined in testing 


testing your team has completed. 


Have you engaged with ACOE for help with | We have followed the advice from ACOE: 
“While the Web Content Accessibility Guidelines (WCAG) do not specifically 


relate to CIVR, | do think three of the overarching principles (perceivable, 
operable, understandable) may be something that could guide you in the design 
phase of the CIVR. | found this great research article that spoke to Accessible 
Conversational User Interfaces that | would encourage you to read through. The 
article discusses more than the scope of your project (CIVR), however, | think it 


testing? 
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brought up some excellent points that could be applied here. | have summarized 
a few of the points below but please take a look at the full article for more 
information. 
e Perceivable: information must be presented to a user in ways they can 
perceive. 
o Interms of CIVR: 
= The ordering of the conversation (e.g. questions asked, 
progression of questions, etc.) should be presented in a 
way that users can follow easily. 
e Operable: users must be able to use the interface and it can’t require an 
interaction that the user can’t perform. 
o Interms of CIVR: 
= The users voice should be understood by the system with 
accuracy 
= Enough time should be provided for users to provide their 
responses — e.g. let the user control the speed and pace of 
the conversation 
= Limit the number of steps required to complete the 
transaction 
e Understandable: users must be able to understand the information 
provided and how to use the interface. 
o In terms of CIVR: 
= There should be a level of predictability (what question 
may come next after a response) 
= Users should be able to get help or tips when needed (if 
they get stuck, etc.) 
For the design aspect of the CIVR, you may be able to engage with a UX expert 
to assist with some of the items above as well. 


Additional Considerations 
e TTY- As discussed, there must be an alternative available for those who 
have hearing or speech disabilities. From our conversation, you 
mentioned that this alternative already exists so that is great. 


Do you have any known accessibility 


challenges? How are you addressing these? 


Have you entered accessibility results in the 


Ontario Websites Database? 


Be agile and user-centred 


[Question Ss Repose 


Nhat the team needs to explore going forwar 


e Plan to incorporate data and user feedback into the project through frequent, iterative 
testing, and use the insight generated to help develop a user-centric product. This feedback 
loop will help the team establish and continuously improve the effectiveness of the CIVR (and 
the features and flows within it) quickly and cost-effectively. 

Connect with the OPS agile community (through its regular drop-ins) to discuss ideas and 


tactics for increasing agil ile = capac ty and applying an agile mindset to the project. 
Review the Service Des 

design as well as Being ei 
As a future roadmap item, the team st "ould explore whether it could also identify health 
related keywords and reroute users to the health queue of the contact centre. This feature is 
likely to enhance customer experience as few users calling to the D&V helpline might also be 
seeking health related support. 


190k for r gui idance on how to complete each phase of service 
ic Service for more ideas on how to work 
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Hope this is helpful! 


How will the CIVR respond to different accents, etc.? Will there be a ‘0’ 
out option to speak to someone directly or an ability to go back to the 
touch tone method? 

Have you reached out to the vendor to determine what accessibility 
features or functions they have already included in the CIVR? 

We would recommend completing some user testing on the system once 
ready. Having a variety of users test the system would provide you with 
some additional insights into the accessibility (and usability) of the 
system before it is launched. 


j» 


It can be on a personal level that something might not work 
If this happens and a client finds a challenge they can speak with 
an agent or will be directed to one 


We can find no reference to this for voice — it appears to be for 
webpages 

If there is direction on this process for voice please advise 

OPS Working Toolkit for Web Accessibility Testing — InsideOPS 


ontario.ca 


This team is prepared to seek feedback and improve the system 
CIVR is a product that learns live and has ongoing learning that will 
require training/tuning 
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How do you work in an agile way (scrum, 
Kanban, hybrid, etc.)? 


For the build of the technology, the team used a sprints approach 

with the vendor 

For content there is Kanban — team daily planner is used 

operationally for issues and trends to watch for. 

e This works well for the distributed SWAT team that supports this 
information; information is sorted in categories for new, backlog, 
follow, closed... 

e Each item has tags to identify the theme and responsibility 

e Individuals are assigned and the system emails any updates to the 
information 

e The Kanban is reviewed in SWATs and daily meetings 


B General Posts files Wiki Issues & Feedback SWAT Notes -+ © Oh Meet 


Issue Content Feedback Content issue Content 
Web; 


eContent New Subject Webpage 


E ove eos. 7) Vahl git vio ace aA GOO- 


@ o © Closing Statements é 7 
one plea to 3 300- sibel fore cn adtest 


E 


Feedback Content | Feedback Flow t New Subject Webpage Gra 


Gs iee eoo 
ordam 
D) Exampl 


eoo. 
Boss 200- 


tent New Subject Process 


Ei AIE Boss O06. onlin sevice se response al 
e © - i Fee tent aaa = 


tent Feedback Content 


ssue Cor 
eee | E Due Q00- 


What agile practices have you adopted 
for this project? E.g., stand-up, user 
feedback sessions, retros, demos/show 
and tells. Tell us how each one is working 


Weekly Stand-ups with the core team now 

Sharing discovery and findings with the core team and 
practice/branch 

Demos/show and tells — Governance Oversight Committee with MTO 
and ServiceOntario, executive, branch, division, partners 

Sprints used with the technology vendor 
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for you. e Communication with Teams, Miro, Sharepoint, and email 


Tell us about your plans to launch your e CIVR technology is complete and tested 
public beta in a production environment. e Recommendations from User Research are being evaluated for 
planning short and long term. Some items are already in backlog for 

this work 

e Content approved and ServiceOntario taking the lead and DEEP is 
supporting 

e The launch is currently planned to be “soft” and for early 
November 2022 

e CIVR will be made available to a TDB% of D&V callers as part of 
iterative implementation plan based on outcomes 

Do you have a prioritized list of features e Yes, there a backlog as this is an MVP 

e They are prioritized by caller value, business value, along with severity, 
and ability to resolve quickly 


in your backlog? How does the list get 


prioritized? 

How will you seek feedback on your MVP e Demonstrations and show and tells will provide more feedback 

while it’s in public beta? e There are opportunities to provide feedback with planned 
surveys 


e Using the reporting to review for information to improve 
e Missed utterances are tracked 


e Trends - Word cloud, answers 


e ServiceOntario offers feedback by email to 
CEO @ServiceOntario.ca 


Once the product leaves beta: This is a new pilot and Product Owner/Manager may change 
e The CIVR is a product of ServiceOntario, Customer Care, Change 
e who will be the product Management and Implementation 


owner/manager? 
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e who from the original team will e Upcoming hiring and transformations with digital, along with 
ServiceOntario making staffing changes to add content experts, chat 


stay on the product? ae 
specialists 


e who on the team will regularly 
Who remains in place? 
e The SWAT team remains in place to resolve issues and content 
e DEEP works in partnership with ServiceOntario and provides 
e what roles are needed to improve regular support 


analyze data and suggest 
improvements? 


and scale the product/service? To regularly analyze data and suggest improvements? 

e howwill your team identify and e ServiceOntario chat team along with DEEP will use data to direct 
improvements — meetings, and support from SWAT 

e Reports can support data changes 

e ServiceOntario has data teams for tracking and analysis 


manage more or less uptake than 
expected? 


Roles are needed to improve and scale the product/service? 
e ServiceOntario product manager 
e Digital Content designers 
e Technology experts 


To identify and manage more, or less uptake than expected? 

e The workforce management team determines the right level of agents 
to have available and will continue to do 

e The work will be based on best practices with a soft launch 
e Advanced reporting will support an efficient review of the chats 

Do you have a plan for QA testing and This is answered above 

e Any new changes or fixes will be tested and include - regression 
testing, accessibility, content, usability 

e Weare able to rollback to previous versions in the CIVR in multiple 
environments 


rolling back changes if need be? 


Tim and Matt 
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How does the team handle decisions for | Content 
e New intents can be added, along with changes to content as 
determined by the team. 
feedback? For example if there a new type of “licence plates” there will be new 
questions that are asked that are not in the CIVR - a new section of 
Tim and Carolyn flows will be added 


design changes that arise from user 


System Design Changes 


e These have been documented and ticketed and are decisioned as to 
the current scope 


How have you documented the build and e DEEP is an enterprise program that provides maintenance and 
support to its clients 
When a ticket is lodged for CIVR, it is directed to CCOPs for triage 
They will look at the issue and determine if it is with the platform 
vendor 
DEEP can assist with AWS support and reachout to vendor 
DEEP has technology specialists and developers on board to 
resolve any platform issues 


maintenance processes for Live? 


Tim and Carolyn 


Use open standards and common platforms 


Question Ci Response S O 
What common technology platforms did AWS is the technology for this iteration — cloud based natural 
; language Al t. This is the essence of the chat conversation 
ou use? Describe how you used them 
r z X Engagement Platform used to host the call queues, which is 


in detail. integrated with the CIVR 
This MVP will be the first of its kind in the Ontario government, and 


will provide learnings that can be leveraged for future CIVR 
that are accessible and have a strong implementations within the OPS. 


No open-source options were used — vendor support 


Did you choose open-source options 


MPBSD 23-071 Record 10 MidBeta-Assessment Final-CIVR-2022-09-26 Page 43 of 52 


developer community supporting them? 


Did you use any existing government We used the current Engagement Platform for voice technology 


Voice-activated IVR applications are being hosted on EP today (eg. 


latf .g., Ontario.ca, One-Key)? If ip 
dppb Agy Onana i) 511, MOF), and this is the next-generation technology. 


not, did you document why not? 


Has your team acquired a new domain Not Applicable for voice 
via Domain.Registation@ontario.ca? 


How have you notified users of any data All users are anonymous that use the CIVR and there is an upfront 
message to notify people to not add any personal information. 

, There have been no decisions on what will be open data at this point, 
and stored and what data will be this will be managed with ServiceOntario Data teams 

available as open data? 


that will be collected, how it will be used 


How will you integrate with any legacy We used the current Engagement Platform for voice technology 
systems, e.g., will you use APIs? Has 

your team developed any new APIs? 

How did you document the API? 


Embed privacy and security by design 


Question Ss Response 


Have you worked with business and e Corporate Security; Information, Privacy and Archives; Cyber security; and 
Internal Audit, were all engaged 

; É j e MPBSD did not require a PIA 

Corporate Security; Information, Privacy e Peter Hope-Tindall — Head of Privacy for MPBSD stated that no PIA 

and Archives; and Internal Audit). How 


information risk teams (for example 
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will you continue to engage these teams Direction/ Recommendation: 
when your product is out of beta? No PIA is required at this time. 


We request that the project explore options to provide a voiced notice of some kind, to 
inform calls that deidentified recordings of their responses will be used to improve the 
accuracy of the system. 


The MGCS Privacy Office concludes that the privacy risks for this proposal appear to be 
LOW at this time. 


However, possible additional privacy risks may be identified as part of the MGCS 
implementation and we direct that the project continue to keep us informed about 
developments and consult with us on any privacy related content or concerns. 


e TRA overall risk profile is categorized as low or very low - June 3, 2022 
e RMR response sent approved and accepted — September 7, 2022 


Describe the measures that your team e User management protocols that are in place today to protect privacy and 
has put in place to ensure overall security will continue to apply. The solution is permission-based and 
requires 2 factor authentication. 
e Call recordings and transcripts are stored in S3 buckets (repository) - these 
buckets will be managed by administrators. 


security of your product and your users. 


Describe your process for understanding e AWS maintains firewalls 
new or ongoing threats. Share your plan e TRA outlines this as a very low risk 
for checking suspicious activity. e Admin access is limited to a few key people 
e Report access is available as read only format 
Describe how the data is being e The solution will not store, collect, use, and disclose sensitive personal 
transmitted and stored. information and project has followed Privacy recommendations from 


MGCS Privacy Office relating to minimizing any risk for inadvertently 
collecting, using, storing personal information 
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Have you completed a TRA? What 
recommendations have you 
incorporated into the service? Are there 
any outstanding recommendations? 
Have you completed pen-testing? 


Do you have a disaster recovery plan? 
Have you shared it with the cluster? 


The solution will use EP web server hosting environment to host the 
technology for CIVR 


Call recordings and transcripts are stored in S3 buckets (repository) - 
these buckets will be managed by administrators. Currently, we have not 
identified storage duration or any purge schedules. 

There will be no change in the collection of personal and/or sensitive 
information in the CIVR, compared to today’s touchtone IVR. 


TRA was completed 


SAS did not recommend this 


Disaster recovery has not changed with this upgrade 

We do have business continuity plans for ON PREM, including geo 
redundancy hosted at KDC 

If the CIVR is not working callers will be transferred to an agent queue 


Support those who need it 


[Question | Response 


Nhat the team needs to explore going forward 

e Team needs to clearly understand what the users are hoping to achieve or looking for from 
the portal (what support are they looking for?). This can only be done by directly talking to 
the end user and conducting qualitative research. Attend user research drop-in sessions to 
get support with developing your user research plan 
The narrative scripts, once written, are hard to change given the scope; next might be to 
integrate health card flow etc. The team should consider how the CIVR scripts could be made 
more user friendly because IVR flow was written with a more sequential flow in mind rather 
than conversational 


What additional supports were added to 


the service to ensure all users were 


Research was completed by ODS for the CIVR 

Callers are looking for a quick and correct answer to their question 
Additional programs and scope will be considered once we have 
captured the results of this pilot with the MVP product 


In a traditional touchtone IVR, the caller is required to ‘listen’ and 
press buttons — hear what they need then hang up or transfer to an 
agent. 
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supported? E.g., call centre, email, changes e With a Conversational IVR, they will be asked to speak their 
intentions or ask their questions. The design should allow for scenarios 
where the caller may require accommodations to perform these actions. 

e Based on research, examples of groups who may experience 
accessibility barriers include: 

e Individuals with hearing impairment 
e Individuals with cognitive disabilities 
e Individuals with speech and language impairments. 

e These groups have been considered during the design of the solution, 
and accessibility accommodation will be incorporated in the call flow 
design. 

e Mitigation strategies include: 

e TTY (Teletype) — callers who have their own TTY device can call 
into ECCS — when the IVR receives TTY calls, they are transferred 
to another phone number received and managed by the OPS 
Enterprise TTY service — where an agent can communicate with the 
caller through text using the OPS TTY software installed on their 
desktop. 

e Our DEEP EP standard is to always 
e provide an out for the callers, therefore saying “Agent” or 

pressing zero is available in the Conversational IVR design. 


e never to “trap” the caller in the IVR whether it is a touch-tone, 
speech or conversational IVR, “Timeouts” can also be used to 
route callers that don’t say or enter anything. There is always an 
exit to the contact centre whether purposefully selected (via 
saying Agent or pressing 0) or unintentionally selected (by 
timing out in a menu, repeating the menu content multiple 
times, or erroring out by saying/pressing something that 
was not expected). 

e Native capabilities within the solution that address accessibility 


barriers. 


to user flow, in-person. 


How have you ensured staff who will be e Planning for staff will be completed with the ServiceOntario team — low 


í impact is expected 
affected by this change are aware and e Whispers will be the same for the agents 
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equipped to provide assistance when e We will ask for any feedback from the agents as well 


necessary? 


Measure performance 


[Question == Repose 


ita paps Oe gng raa nds aah PAE IRE The KPIs have been baselined and the reports for MVP created 
. Digital Performance Measurement resources i j r i Fi Š z ž 
overall outcomes. e The basic metrics will be reported frequently in the first few weeks and 


Team needs to define what success means to users versus the business. What may be a user 


goal assumed by the business may not in reality be what users are trying to achieve. For will be a regular part of the training and tuning process 
example, a user's question may be answered through CIVR but the issue could still remain e ServiceOntario will be able to build this into their regular reporting 


unresolved. This does not necessarily mean success for the user or even the portal. 


Conduct a customer sentiment analysis to understand callers’ experience with CIVR. e Call listening was found to be effective in understand the call patterns 

Team suggested that they would review the metrics on a monthly basis. We encourage the ` 

team to also connect metrics with the executive dashboard with the ability to show and importance 

ii ia i The ServiceOntario Customer Experience Office also compiles 
feedback from the public regarding contact centre services, including 


the telephone systems. This feedback may be filtered by program 


What are your KPIs for this service? Share 
survey. 


some of the quantitative and qualitative 


metrics. 
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Share baseline data for this service. How will 
you collect relevant data and report on 
performance? 


Have you embedded Google Analytics or 
equivalent to capture key usage stats and 


user funnels/flows? 


KPI Measures 
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(GSIC 


Success is about more than delivering a project on time and on budget- it is about 
meeting our callers need for service, both to selfserve and to reach an agent. 


* calleris asked if they are willing to participate in a survey 
* if they agree they are able to confirm their phone number 
* The system places an outbound call to the caller to conduct the survey 
CIVR — Relaunch Survey 
* Callback to callers after the call (80% Standardapprovals required 
* CIVR tool to connect to be devel 
Speed to need Baseline- time for callers to reach first topic in IVR 
(Time to first CIVR - time for callers to reach first topic 
topic) * compare against baseline to measure improvementi§%) 
Baseline- time for callers to reach MMR in IVR by pressing 0 
Chae > MAY CIVR - time for callers to reach MMR 
* compare against baseline to measure improvemeni@%) 
Baseline- measured in IVR (hangup /total callers) 
Ehan . “hangup” in a topic will be considered a setfrve 
(% hang uPS) ciyp_ Aspirational it would increase, but success is maintainingQ%) 
Request Agent Baseline- number of callers thapress 0 from top menu/total calls 
(% top menu) CIVR - number of callers thafsk to speak to an agent immediatelyi0%) 


GovTechON Ontario Y 


There is a Baseline above for KPI Measures 
Projection for volume metrics will no be as relevant with a staggered 
implementation 


The best understanding of usages came from reviewing the 800 
calls and the 1400 IVR surveys. 

Future knowledge will be from the reporting in the system 
ServiceOntario has a Google dashboard ServiceOntario 
Ontario.ca Analytics Dashboard » Top ServiceOntario Pages 
(google.com) 

Authoured by MPBSD Communications provides detail on high use 
webpages & ServiceOntario uses Google analytics on the Chat team 
and in Business Intelligence, and DEEP has access as well 

CMI the ServiceOntario team responsible for content has Google 
access for MPBSD and MTO web 
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USING THIS DASHBOARD: LEGEND: 


User Demographics 


Who are our users? © Where are users located? What devices are being used? 
City Users ~ 


What language do users set their browsers to? 
Language Users ~ 


35-44 45-54 55-64 Bt erdo = 


How are users visiting ServiceOntario pages?® What time of day has the most visitors? © How many users visited ServiceOntario pages 
for the first time? 


Se O88 


Sep 6, 2022 - Sep 12, 2022 


2AM - De a 
i H 
3508 
2207 
80 orate 


auas 


How many users clicked to renew their 
ServiceOntario products? 


—~____ 304989 


Sep 6, 2022 - Sep 12, 2022 


Blue 'Renew’' Button Clicks 


What are the most used Blue "Renew" Buttons on ServiceOntario pages? How do different age groups interact with Blue Buttons? © 
sos LALL 
27386 ll axe L l. > 
How are you measuring the offline service (if e This service is anonymous 


e The current EP (ice) system allows for confidential investigations 

e The assumption is people find the phone number online — with the 
moving from the offline service to the online majority from the “contact us pages” for D&V information 

e TBD% will be offered the CIVR with a staggered implementation 


there is one), and how do you track people 
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one? e There is a whisper for agents is the same with driver and vehicle and 
they will get the correct experience with the agent 
e We compare the initial listening to the IVR to ongoing as a measure — 
if there are deviations from the norm we can investigate why 
e QuickSight dashboard example 


datetimeparsed (Week) 
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Be a good data steward 


Question =i Response 


Oe ee e The top things asked were identified 


è Team could look into collecting and publishing anonymous data such as . m 
i Top things people are asking for e By listening to over 800 calls 
Top phrases or keywords people are using : 
Top topics being asked by the callers that CIVR is unable to answer e Review of the chatbot questions 
Reports will identify topics in the future and phrases/words that it is unable 
to answer 
What are the unique data holdings Currently, ServiceOntario wait times and call volumes (contact centres) 
are published to the Ontario data catalogue: ServiceOntario wait times and 


? . 
generated by oo, product: How is data call volumes (contact centres) - Datasets - Ontario Data Catalogue. This 


used and managed? includes D&V data. 
Reports - outlined previously 
e Used for monitoring and improvement 
Surveys for IVR and Agent 
e Used to understand satisfaction, barriers and caller expectations 
e Will not identify the callers or numbers 


What data will be published as Open ServiceOntario has its own data teams RIM and BI 

Data? Currently, ServiceOntario wait times and call volumes (contact centres) 
are published to the Ontario data catalogue: ServiceOntario wait times and 
call volumes (contact centres) - Datasets - Ontario Data Catalogue. This 
includes D&V data. 
This data will continue to be published 
MPBSD data leads are in corporate 
e Executive lead - Shawn.Lawson@ontario.ca 
e Manger lead - Mona.Lee@ontario.ca 
e Staff lead - Sabrina.Mughal@ontario.ca 


Has your team identified any other We deal with 44 contact centres and they want to understand similar things 
ministry programs that are currently We host two other voice recognition services (MOF, 511) 
CIVR is new unique 
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collecting similar data from Ontarians? 
Please share details. 


How does your product share data in e ICE Reporting feeds to ServiceOntario 
machine readable formats? e CIVR reports will be available to Customer Care and Reporting areas 


